" Input: { (repo_uoa) - program repo UOA (module_uoa) - program module UOA (data_uoa) - program data UOA or (program_uoa) - useful if univeral pipeline is used, i.e. ck run pipeline:program program_uoa=... or taken from .cm/meta.json from current directory (random) - if 'yes', random selection of program, cmd, dataset uoa and dataset file (to support collaborative program optimization) (skip_local) - if 'yes', skip detection of program in a local path (program_tags) - select programs by these tags (program_dir) - force program directory (target) - target machine added via 'ck add machine' with prepared target description (useful to create farms of machines for crowd-benchmarking and crowd-tuning using CK) (host_os) - host OS (detect, if omitted) (target_os) - OS module to check (if omitted, analyze host) (device_id) - device id if remote (such as adb) (local_platform) - if 'yes', use host_os/target_os from the current platform (useful when replaying experiments from another machine and even OS) (prepare) - if 'yes', only prepare setup, but do not clean/compile/run program (save_to_file) - if !='', save updated input/output (state) to this file (skip_interaction) - if 'yes' and out=='con', skip interaction to choose parameters (skip_device_init) - if 'yes', skip device init (skip_info_collection) - if 'yes', skip info collection (skip_device_info) - if 'yes', skip any device info - useful to prepare experiment crowdsourcing packs for remote devices Pipeline sections' settings: (compute_platform_id) - if !='', set env['CK_COMPUTE_PLATFORM_ID'] (compute_device_id) - if !='', set env['CK_COMPUTE_DEVICE_ID'] (no_platform_features) - if 'yes', do not collect full platform features (no_dataset_features) - if 'yes', do not search and extract data set features (no_clean) - if 'yes', do not clean directory before compile/run (no_compile) - if 'yes', do not compile program (useful when running the same program under different system state conditions: CPU/GPU freq, cache/bus contentions, etc) (compile_only_once) - if 'yes', compile only at first iteration (no_compiler_description) - if 'yes', do not search for most close compiler description with flags ... (no_run) - if 'yes', do not run program useful when using autotuning to find bugs in compiler, or find differently generated code sequencies, etc ... (no_state_check) - do not check system/CPU state (frequency) over iterations ... (generate_rnd_tmp_dir) - if 'yes', compile and run program in randomly generated temporal dir or (tmp_dir) - if !='', use this tmp_dir (skip_clean_after) - if 'yes', do not remove run batch (keep) - the same as skip_clean_after (console) - if 'yes', output from program goes to console rather than file (usually for testing/demos) (cmd_key) - CMD key (cmd_keys) - Select only from this list of available CMD keys (dataset_uoa) - UOA of a dataset (dataset_file) - dataset filename (if more than one inside one entry - suggest to have a UID in name) (extra_dataset_tags) - list of extra data set tags (useful to set "small" during mobile phone crowdtuning) (compiler_env_uoa) - env of a compiler (compile_type) - static or dynamic (dynamic by default; however takes compiler default_compile_type into account) or (static or dynamic) (compiler_description_uoa) - compiler description UOA (module compiler), if not set, there will be an attempt to detect the most close by version (compiler_vars) - dict with set up compiler flags (-Dvar=value) -> they will update the ones defined as default in program description ... (no_vars) - skip compiler vars (if you want to use default ones from the sources) ... (remove_compiler_vars) - list of compiler vars to remove (extra_env_for_compilation) - set environment variables before compiling program (flags) - compile flags (lflags) - link flags (compiler_flags) - dict from compiler description (during autotuning), if set, description should exist in input:choices_desc#compiler_flags# ... (best_base_flag) - if 'yes', try to select best flag if available ... (speed) - the same as above (skip_best_base_flag) - if 'yes', do not use best base flag (useful for exploration of other levels -O2,-O1,etc) (env_speed) - use environment flag for best optimization (CK_OPT_SPEED) (shared_solution_cid) - CID-UID1-UID2 of the shared optimization solution at cKnowledge.org/repo You can find it by clicking on a "Copy CID to clipboard" button of a given solution. See example at http://cknowledge.org/repo/web.php?wcid=8289e0cf24346aa7:79bca2b76876b5c6 27bc42ee449e880e:79bca2b76876b5c6-8289e0cf24346aa7-f49649288ab0accd (Ocid-uid1-uid2) Substituting compiler -Ox levels with shared solutions in above format (-O27bc42ee449e880e:79bca2b76876b5c6-8289e0cf24346aa7-f49649288ab0accd) (select_best_base_flag_for_first_iteration) - if 'yes' and autotuning_iteration=0 (env) - preset environment (env.{KEY}) - set env[KEY]=value (user-friendly interface via CMD) (deps.{KEY}) - set deps[KEY]["uoa']=value (user-friendly interface via CMD to set any given dependency) (preset_deps) - dict with {"KEY":"UOA"} to preset dependencies (params) - dictionary with parameters passed via pre/post processing to third-party tools for example, to configure ARM Workload Automation (params.{KEY}) - set params[KEY]=value (user-friendly interface via CMD) (extra_env) - extra environment as string (extra_run_cmd) - extra CMD (can use $#key#$ for autotuning) (debug_run_cmd) - substitute CMD with this one - usually useful for debugging to pre-set env for all deps (run_cmd_substitutes) - dict with substs ($#key#$=value) in run CMD (useful for CMD autotuning) (sudo) - if 'yes', force using sudo (otherwise, can use ${CK_SUDO_INIT}, ${CK_SUDO_PRE}, ${CK_SUDO_POST}) (affinity) - set processor affinity for tihs program run (if supported by OS - see "affinity" in OS) examples: 0 ; 0,1 ; 0-3 ; 4-7 (the last two can be useful for ARM big.LITTLE arhictecture (repeat) - repeat kernel via environment CT_REPEAT_MAIN if supported (do_not_reuse_repeat) - if 'yes', do not reuse repeat across iterations - needed for dataset exploration, for example (skip_calibration) - if 'yes', skip execution time calibration (otherwise, make it around 4.0 sec) (calibration_time) - calibration time in string, 4.0 sec. by default (calibration_max) - max number of iterations for calibration, 10 by default (statistical_repetition_number) - current statistical repetition of experiment (for example, may be used to skip compilation, if >0) (autotuning_iteration) - (int) current autotuning iteration (automatically updated during pipeline tuning) (the_same_dataset) - if 'yes', the dataset stays the same across iterations so skip copying dataset to remote from 2nd iteration (repeat_compilation) - if 'yes', force compilation, even if "statistical_repetition_number">0 (cpu_freq) - set CPU frequency, if supported (using SUDO, if also supported) using script ck-set-cpu-online-and-frequency if "max" - try to set to maximum using script ck-set-cpu-performance if "min" - try to set to minimum using scrupt ck-set-cpu-powersave (gpu_freq) - set GPU frequency, if supported (using SUDO, if also supported) using script ck-set-gpu-online-and-frequency if "max" - try to set to maximum using script ck-set-gpu-performance if "min" - try to set to minimum using scrupt ck-set-gpu-powersave (energy) - if 'yes', start energy monitoring (if supported) using script ck-set-power-sensors (gprof) - if 'yes', use gprof to collect profile info (perf) - if 'yes', use perf to collect hardware counters (vtune) - if 'yes', use Intel vtune to collect hardware counters (sim) - if 'yes', use architecture simulator (found by tags "arch","sim") (valgrind) - if 'yes', add valgrind (dvdt_prof) - if 'yes', run program under dividiti's OpenCL profiler (mali_hwc) - if 'yes', attempt to extract MALI GPU hardware counters (milepost) - if 'yes', attempt to extract static program features using Milepost GCC and cTuning CC (milepost_out_file) - if !='', record extracted MILEPOST features to this JSON file (compile_timeout) - (sec.) - kill compile job if too long (run_timeout) - (sec.) - kill run job if too long (post_process_script_uoa) - run script from this UOA (post_process_subscript) - subscript name (post_process_params) - (string) add params to CMD (dependencies) - compilation dependencies (deps_cache) - cache with resolved dependencies for reuse (if needed) (reuse_deps) - if 'yes' reuse dependencies (force_resolve_deps) - if 'yes', force resolve deps (useful for crowd-tuning) (choices) - exposed choices (if any) (choices_order) - vector of flattened choices (useful if optimizations need order such as LLVM or using our GCC plugin iterface to reorder passes, since 'choices' dict does not have order) (features) - exposed features (if any) (characteristics) - measured characteristics/features/properties (if any) (state) - kept across pipeline iterations (for example, during autotuning/exploration) (tmp_dir) - if temporal directory is used, return it (useful if randomly generated, to be reused for run or further iterations) (repeat) - kernel repeat ... (features.platform.cpu) - CPU features/properties obtained during iterations to check that state didn't change ... (skip_output_validation) - skip validation of output (dangerous during auto-tuning - some optimizations may break semantics or change accuracy) (output_validation_repo) - output validation repo UOA (when recording new output) (program_output_uoa) - use this UOA to check/record program output (to have the same output entry for groups of similar programs) (overwrite_reference_output) - if 'yes', overwrite reference output (useful if broken) (quiet) - if 'yes', automatically provide default answer to all questions when resolving dependencies ... (last_md5) - if !='', check if MD5 and fail if didn't change! (last_md5_fail_text) - to recognize that pipeline failure is not really a failure, but MD5 is the same (useful when pruning compiler flags found during collaborative autotuning, particularly via mobile devices (less time to prune results)) (skip_print_timers) - if 'yes', skip printing fine-grain timers after execution (add_rnd_extension_to_bin) - if 'yes', add random extension to binary and record list (add_save_extension_to_bin) - if 'yes', add '.save' to bin to save during cleaning ... (install_to_env) - install dependencies to env instead of CK-TOOLS (to keep it clean)! (safe) - safe mode when searching packages first instead of detecting already installed soft (to have more deterministic build) (skip_exec) - if 'yes', do not clean output files and skip exec to be able to continue post-processing during debuging } Output: { return - return code = 0, if successful > 0, if error (error) - error text if return > 0 !!! The same as input, but with updated information !!! ready - if 'yes', pipeline is ready (all obligatory choices are set) if 'no', clean/compile/run program is postponed state - should be preserved across autotuning, active (online) learning, exploration, validation iterations } "